Gustavo Diaz
McMaster University
gustavodiaz.org
diazg2@mcmaster.ca
Slides: talks.gustavodiaz.org/tec
Explicit: Is a little bias worth the increase in precision?
Implicit: Improving precision without sacrificing unbiasedness?
Explicit: Is a little bias worth the increase in precision?
Implicit: Improving precision without sacrificing unbiasedness?
Here is a list of things that some people have done.
Please listen to them and then tell me HOW MANY of them you have done in the past two years.
Do not tell me which ones. Just tell me HOW MANY:
Do not tell me which ones. Just tell me HOW MANY:
Do not tell me which ones. Just tell me HOW MANY:
\[ \text{Proportion(Voted yes)} =\\ \text{Mean(List with sensitive item)} -\\ \text{Mean(List without sensitive item)} \]
Did you vote YES or NO on the Personhood Initiative, which appeared on the November 2011 Mississippi General Election Ballot?
\[ \text{Proportion(Voted yes)} =\\ \text{Mean(Voted yes)} \]
List A
List B
Organization X (advocating for immigration reduction and measures against undocumented immigration)
Randomly appears in list A or B
Single list: Half of the respondents see sensitive item
Double list: Everyone sees it
Equivalent to two parallel list experiments
\[ \hat{\tau}_A = \text{Mean}(A_t) - \text{Mean}(A_c) \]
\[ \hat{\tau}_B = \text{Mean}(B_t) - \text{Mean}(B_c) \]
\[ \hat{\tau}_{Pooled} = (\hat{\tau}_A + \hat{\tau}_B)/2 \]
Baseline lists need to be comparable
Easiest way is to use paired items
American Family Association (A) \(\approx\) Tea Party Patriots (B)
BUT that makes it easier to spot the sensitive item
| List order | Sensitive item location |
|---|---|
| Fixed | Fixed |
| Randomized | Fixed |
| Fixed | Randomized |
| Randomized | Randomized |
| List order | Sensitive item location |
|---|---|
| Fixed | Fixed |
| Randomized | Fixed |
| Fixed | Randomized |
| Randomized | Randomized |
| List order | Sensitive item location |
|---|---|
| Fixed | Fixed |
| Randomized | Fixed |
| Fixed | Randomized |
| Randomized | Randomized |
| List order | Sensitive item location |
|---|---|
| Fixed | Fixed |
| Randomized | Fixed |
| Fixed | Randomized |
| Randomized | Randomized |
| List order | Sensitive item location |
|---|---|
| Fixed | Fixed |
| Randomized | Fixed |
| Fixed | Randomized |
| Randomized | Randomized |
Design effect (Blair and Imai 2012)
The inclusion of a sensitive item affects how survey participants respond to the baseline items within the list.
Carryover design effect
The inclusion of a sensitive item in one list affects how participants respond to the baseline items in the other list.
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
Goal: Detect asymmetric shift across treatment schedules
Two tests:
Difference-in-differences (clustered responses)
Signed-rank test (paired responses)
Goal: Detect asymmetric shift across treatment schedules
Two tests:
Difference-in-differences (clustered responses)
Signed-rank test (paired responses)
\[ \hat{\tau}_1 = \text{Mean}(\text{First list}_t) - \text{Mean}(\text{First list}_c) \]
\[ \hat{\tau}_2 = \text{Mean}(\text{Second list}_t) - \text{Mean}(\text{Second list}_c) \]
| Experiment | Statistic | p-value |
|---|---|---|
| Organization X (advocacy group) | 0.079 | 0.623 |
| Organization Y (border patrol) | -0.268 | 0.082 |
| Experiment | Statistic | p-value |
|---|---|---|
| Organization X (advocacy group) | 0.079 | 0.623 |
| Organization Y (border patrol) | -0.268 | 0.082 |
| Experiment | Statistic | p-value |
|---|---|---|
| Organization X (advocacy group) | 0.079 | 0.623 |
| Organization Y (border patrol) | -0.268 | 0.082 |
Criminal governance
Informal tools to control community behaviors that involve political, economic, or social aspects with the goal of profiting from illicit markets.
More common in contexts of high violence and low state presence
Why do we see them in Uruguay?
Goal: Estimate extent of criminal governance tools
Facebook sample of Montevideo residents (N = 2688)
Four criminal governance strategies
Facebook sample of Montevideo residents (N = 2688)
Four criminal governance strategies
Things people have experienced in the last six months:
| List A | List B |
|---|---|
| Saw people doing sports | Saw people playing soccer |
| Visited friends | Chatted with friends |
| Activities by feminist groups | Activities by LGBTQ groups |
| Went to church | Went to charity events |
Things people have experienced in the last six months:
| List A | List B |
|---|---|
| Saw people doing sports | Saw people playing soccer |
| Visited friends | Chatted with friends |
| Activities by feminist groups | Activities by LGBTQ groups |
| Went to church | Went to charity events |
| Gangs threatening neighbors | Did not drink mate |
Things people have experienced in the last six months:
| List A | List B |
|---|---|
| Saw people doing sports | Saw people playing soccer |
| Visited friends | Chatted with friends |
| Activities by feminist groups | Activities by LGBTQ groups |
| Went to church | Went to charity events |
| Did not drink mate | Gangs threatening neighbors |
Placebo item more frequent than we anticipated
Offsets prevalence rates we would have observed
Solution: Reconstruct estimate bounds without placebo item
Problem: Respondents may have noticed sensitive item and altered responses in unintended ways
Goal: Rule out strategic errors
Standard (Blair and Imai 2012):
No liars (one-sided lying)
No design effects
Standard (Blair and Imai 2012):
No liars (one-sided lying)
No design effects
Standard (Blair and Imai 2012):
No liars (one-sided lying)
No design effects
Standard (Blair and Imai 2012):
No liars (one-sided lying)
No design effects
| Sensitive item | Statistic | p-value |
|---|---|---|
| Threaten neighbors | 0.12 | 0.41 |
| Evict neighbors | 0.08 | 0.58 |
| Make donations | -0.24 | 0.16 |
| Offer work | -0.11 | 0.47 |
| Sensitive item | Statistic | p-value |
|---|---|---|
| Threaten neighbors | 0.12 | 0.41 |
| Evict neighbors | 0.08 | 0.58 |
| Make donations | -0.24 | 0.16 |
| Offer work | -0.11 | 0.47 |
Standard (Blair and Imai 2012):
No liars (one-sided lying)
No design effects
Implications:
Standard (Blair and Imai 2012):
No liars (one-sided lying)
No design effects
Additional:
Standard (Blair and Imai 2012):
No liars (one-sided lying)
No design effects
Additional:
Known placebo proportion
Standard (Blair and Imai 2012):
No liars (one-sided lying)
No design effects
Additional:
Known placebo proportion
Standard (Blair and Imai 2012):
No liars (one-sided lying)
No design effects
Additional:
Known placebo proportion
TBD
Placement
|
||
|---|---|---|
| List A | List B | |
| Sensitive item | ||
| Organization X | 545 | 525 |
| Organization Y | 537 | 543 |
\[ \underbrace{\widetilde{T} = \sum_{i=1}^N \text{sgn} \{(z_i - (1-z_i)) (Y_{i1} - Y_{i2})\} \times \tilde{q}_i}_{\text{Sum of signed ranks } \tilde{q}_i} \]
\[ \tilde{q}_i = {q_i-1 \choose m-1} \text{ for } q_i \geq m\text{; } \tilde{q}_i = 0 \text{ for } q_i < m \]